The pitfalls of questionnaire surveys
Practical Action
Administering: When administering a survey the meaning can be lost in translation;
not enough time is put into training; administrators can show bias or ask leading
questions therefore influencing the results; there may not be gender equality within
your administering team; lack of confidentiality in cases where information is
sensitive, especially issues surrounding health. The administering process is also a
drain on resources, time and money needed to carry them out.
Analyzing: This can be time consuming and can also be a drain on resources. Often
in the beginning thought is not put into what the information is going to be used for
and how it is going to be analyzed. How do you identify the most crucial information,
how are you going to generate your results and how do you plan to present this data?
How much of the data collected will be used?
Beyond questionnaires
There are many different methods that can be applied as a replacement or in combination
with a questionnaire survey. A Participatory Rural Appraisal (PRA) is one such method where
the community identifies the issues and is guided by a facilitator. This is a social approach to
gathering data which is flexible, informal and performed jointly with the community tapping
into local knowledge and experience. A PRA may include and is not limited to the following
tools: (Handbook, ILO Assist)
Key informant interviews: The interviewee raises issues and comments within a loose
framework. Individuals can directly identify the most important issues faced by the
community.
Focus group sessions: Open-ended questions can be asked and discussion amongst
community members can occur. The community is given the freedom to contribute.
Transect walks: Mixed groups of professionals and community members walk
through an area to observe and discuss.
Community mapping: Members of the community develop their own map of the area
and reflect on what is important. This can provide spatial information such as land
use and ownership. This can be achieved through mapping on the ground whilst in
the field, on paper or using GIS mapping tools.
Diagrams and modeling: Both these tools present information in a simplified form
which enables analysis, facilitates communication and stimulates discussion eg
timelines, seasonal calendars, linkage diagrams.
Preference: Ranking or scoring by putting in order of preference highlights the
priorities of the community. For example wealth ranking is based on a list of
indicators set by the community where individuals sort cards of households into piles
according to their wealth. An average is then calculated. Other tools include matrix
ranking and scoring.
If you have concerns about getting quantitative data from participatory methods you can
employ triangulation. Through a participative process, people from various professions,
community members with different views and experience can work together and use a
combination of visuals and tangibles, for example maps/diagrams and models/counters
(stones, beans etc). It is possible however to derive ‘participatory numbers’ from activities
such as counters, calculating, piling, scoring, ranking, estimating and valuing (2007
Chambers: working paper 296, pp 11).
Rapid Rural Appraisal (RRA) is a method still largely researcher based and passive in nature.
It is however a quick method that involves semi-structured interviews and informal checklists.
PRA seems to have taken its place (even though PRA is mainly sourced from methods of
RRA) and is considered to be more empowering instilling a sense of ownership over the
process. As with the questionnaire survey the information obtained from a RRA is analyzed
back in the office. Participatory processes (PRA) are analyzed by the community whilst you
are in the field, once again providing an integrated community approach.
3